No, you're not a coward.
Here's an example of what I (with the help of my wife) did to end this kind of nonsense with my toxic mother:
So years back (still an active, believing witness elder), I would get weekly calls from my mother where she'd push my buttons. I'd hang up, feeling angry, guilty, upset -- and this got my wife mad (don't get my wife mad!)
So my wife wisely suggested that next time I get my call, put the speakerphone on and tell mother "I have you on speaker and Mrs BOC is listening."
Well, wouldn't you know, next call comes, and I do exactly what my wife suggested. After telling my mother I had her on speaker, she seemed to forget why she called and asked about the weather or something like that. The exchange ended quickly and without any of the usual nonsense.
I repeated this several weeks in a row, and guess what? Same thing. Eventually, calls from my mother ceased altogether. Now, I get a call from her once in a blue moon, and when she does call, she is on her "best behavior" even without my wife listening in.
Why did this work? Because by putting my mother on speaker with my wife, my mother's "millieu control" was compromised. Controlling, abusive people don't like to appear controlling and abusive, so they do what they do out of sight/earshot or in secret. Taking away control of the "millieu" (in this case a private phone conversation) was enough to quash the controlling behavior, and over time it just ended.
So if you can find some small but specific way of taking the control out of your mother's hands (and you would know how to do this better than me) it may just work.